Repository navigation
Require an AI cross-review before merge - #229
lwwmanning wants to merge 16 commits into
Conversation
Every pull request needs a Codex review of its head commit. A pull request whose footer or author names Codex also needs a Claude review. The gate sets the commit status "AI cross-review" for the main ruleset to require. Signed-off-by: Will Manning <will@willmanning.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
To use Codex here, create an environment for this repo. |
…view-f2ab62 Signed-off-by: Will Manning <will@willmanning.io> # Conflicts: # CLAUDE.md
CodeQL flagged the explicit checkout of the head commit as an untrusted checkout, because the workflow also runs on issue_comment. Signed-off-by: Will Manning <will@willmanning.io>
Codex Review Summary: 🔄 RunningThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
CodeQL flags any job that runs the pull request's code in a workflow that also runs on issue_comment. ai-review.yml now runs on pull_request only, and ai-review-recheck.yml reruns main's copy of the gate on comments and reviews. The gate's token also gets pull-requests: write, which a comment on a pull request needs. Signed-off-by: Will Manning <will@willmanning.io>
|
@codex review |
🔦 Lighthouse audit
Run |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e51dd80774
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex's review found that the gate ran in the pull request's context: forks and Dependabot PRs got a read-only token, a PR could change the script that judged it, and fork PRs could never get a Claude review. The workflow now runs on pull_request_target, comments and reviews, and every job checks out main. Claude reads the PR through the API. The gate also asks Codex again after 20 minutes without a start (three times at most), retries a Claude review that ended without a verdict, and ignores dismissed Codex reviews. ci.yml runs the gate's tests. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 25fbcceb11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The gate sets itself to pending before it reads the reviews, so a failed run cannot leave an old success in place. It fails when another open PR has the same head commit, because a commit status counts for every PR on that commit. A submitted Codex review of the head now counts even while the summary comment still says the review is running. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20e6373921
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The Claude status name now holds the PR number, so a review of one PR does not count for another PR on the same commit. The gate counts Claude's blocking findings from its unresolved P0 and P1 threads, as it does for Codex, so resolving a thread and commenting clears it without a new push. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5093392b2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A PR that moves to another base branch keeps its head commit, but its diff can change, so the gate posts a marker and fails until a new commit arrives. Only P0 and P1 threads count as evidence for Claude's counted findings. The gate reads every page of the commit's statuses, and counts only marker comments from github-actions[bot]. Signed-off-by: Will Manning <will@willmanning.io>
…view-f2ab62 Signed-off-by: Will Manning <will@willmanning.io> # Conflicts: # CLAUDE.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3516cffac1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Drop the pull_request_review trigger, which runs the PR's copy of the workflow. Claude reviews only PRs by authors with write access; an outside PR that Codex helped write needs a maintainer's approval of the head commit. Record a base change as a commit status before the notice comment. Mark a Claude review as started only after setup, and retry a review that stays pending for 45 minutes. Check at verdict time that each blocker Claude counts has an open thread. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d46d93d67d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
author_association calls a read-only organization member a MEMBER, so the gate now asks the permission API whether the author, or an approver of the head commit, can push. The Claude action can check out the PR's branch, so the job checks out main again before it records the verdict, and Claude reads CLAUDE.md from main. The gate's rerun after a review is now a job in the gate's concurrency group. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 576ce296d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…odex's finding flag The first gate run on a head commit records the base branch as a status, and later runs compare against it, so a cancelled waiting run cannot lose a base change. An approval counts only while it is the reviewer's latest approval or request for changes. When Codex reports findings on the head commit but no Codex thread on that commit shows one, the gate fails. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 92517a3682
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A commit status belongs to the commit, so another PR with the same head would inherit it. The gate now runs on closed and sets its status back to pending unless the PR merged. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d313911250
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const shown = input.threads.some( | ||
| (t) => isCodexThread(t) && t.commitSha === input.headSha | ||
| ); | ||
| if (reported && !shown) { |
There was a problem hiding this comment.
Detect partially missing Codex finding threads
When Codex reports multiple findings but only one inline thread is published, shown becomes true even if another missing finding is P0/P1. The fresh partial-publication case is not covered by the prior no-thread fix: the visible thread can be resolved or P2/P3, after which the gate publishes success while the missing blocker has no resolvable thread. Preserve enough structured finding information to detect incomplete publication rather than accepting any single thread as evidence for all findings.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changed. Codex's summary gives only booleans (has_any_findings, has_unresolved_findings), not a count, so the gate cannot tell how many threads to expect. Codex posts its inline comments as part of one review: on this PR, each Codex review holds all of its comments (from 1 to 7). That suggests a review is published whole or not at all, which the no-thread check covers. I'm leaving this thread open for a maintainer to decide.
🤖 Addressed by Claude Code
| permissions: | ||
| contents: read | ||
| # Set the `AI cross-review` status. | ||
| statuses: write | ||
| # Post `@codex review` when Codex has not reviewed the head commit. A |
There was a problem hiding this comment.
Prevent pull-request workflows from forging the gate status
The required result is a generic commit status written by the GitHub Actions app, so a same-repository contributor can modify an existing pull_request workflow such as ci.yml or lighthouse.yml, request statuses: write, and POST AI cross-review: success for their own head SHA without satisfying this gate. GitHub documents that pull_request workflows run against the PR merge branch and that workflow authors can change GITHUB_TOKEN permissions, including statuses: write; pinning the required status to GitHub Actions therefore does not distinguish the trusted base workflow from PR-controlled workflows. Publish the result through a dedicated app/source that PR workflows cannot impersonate, or enforce this as a required workflow instead. GitHub pull-request event semantics GitHub token permissions
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changed. This is a real limit of a commit status: in this repository, any pull_request workflow that a writer edits could post one. Fork PRs get a read-only token, so they cannot. The fix is to publish the result from a dedicated GitHub App, and pin the ruleset's required check to that app. Creating an app is a setup step for a maintainer, so I'm leaving this thread open for a maintainer to decide.
🤖 Addressed by Claude Code
The gate job sets its status to pending before checkout and setup, so a failed setup cannot leave an earlier pass. A closed PR, merged or not, withdraws its status. The Claude job records a verdict only when main was checked out again. The base record holds the branch name's SHA-256, so a long name cannot be cut to match another. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37b41373b0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A review, or a PR closing beside a sibling with the same head, starts no gate run. Every 15 minutes a scheduled run from main evaluates each open, ready PR again under the gate's lock, and writes the status only when the result changed. A Codex review still in progress after an hour counts as stopped, so the gate may ask for it again. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc31d561a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…reviews The refresh starts the workflow by workflow_dispatch for a PR whose Claude review is due, because only a gate run can start the Claude job. The gate starts at most three Claude reviews per commit. A dispatched run has no PR context for the inline comment tool, so Claude can post findings through the review comments API. A dismissed Codex review of the head commit no longer counts through Codex's summary comment. Signed-off-by: Will Manning <will@willmanning.io>
What changes
This adds a merge gate for AI review to both sites' pull requests, in a new workflow,
ai-review.yml. The other checks stay as they are.Before, Codex reviewed some pull requests on its own, and Claude reviewed only when someone tagged
@claude. Neither review blocked a merge. After, the gate sets the commit statusAI cross-reviewon the head commit. Every pull request needs a clean Codex review of that commit. If the footer or the author names Codex, the pull request also needs a clean Claude review, which the workflow runs.A P0 or P1 finding fails the gate, so auto-fix can act on it. When Codex has not reviewed the head commit, the gate posts
@codex review. Renovate and Dependabot pull requests pass without review.After merge, the
mainruleset must requireAI cross-reviewbefore the gate blocks anything.How it was checked
bun test scripts/ai-review-gate.test.ts, the typecheck andbun run check:cipass. Dry runs against #216, #220, #225, #206 and #227 gave the expected status for each.git commit -s), for the DCO check.bun run check:cipasses.REUSE.toml.🤖 Generated with Claude Code